DOCUHENT BSSQH2 



ED 131 103 



Tfl 005 827 



ADTHOH 
TITLE 

NOTE 

EDBS PBICE 
DESCRIPTORS 

IDENTIFIERS 



Behm, Eobert J. ; Schill^ Hilliam J, 

The Examination of Intra^-group Agreement of Q-Sort 

Hesponses • 

9 p. 

HF^$0.83 HC-$1.67 Pins Postage. 
♦Correlation; Matrices; *Q Sort; *Statistical 
analysis; *Test Reliability 

Kendall Coefficient of Concordance; Spearman Rank 
Order Correlation; Statistical Package for Social 
Sciences 



ABSTRACT 

A technique for 
Q-sorts of two or more groups o 
on the relationship between the 
(H) and the Spearman rank order 
statistical treatment of Q-sort 
intercorrelations rather than a 
provides insight into the level 
agreement and permits the use o 
programs. (Author/flV) 



assessing the agreement between the 
f subjects is presented which relies 
Kendall coefficient of concordance 
correlation (rho) . The proposed 
data involves the use of a number of 
direct computation of W. This method 
and nature of the intra-group 
f certain readily available computer 
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^ The Q-sort technique was developed in the 1930's by Stephenson and Thomson. 

^ A comprehensive analysis of the technique was first published in the 1950's 

(Stephenson, 1953). The Q-sort technique is essentially, a card-sort method for 
ordering statements, objects or adjectives by rank. This technique has been 
successfully used in education (Schill, 1961), psychology (Block, 1961) and 
communications (Emmert and Brooks, 1970). Redbum (1975) indicated the ad- 
vantages of this method in comparison to normative measures and movement of 
subjects relative to a priori scales. 

Many of the studies to date have examined the responses of two or more 
selected subjects. However, the experimenter may be interested in a comparison 
of the Q-sorts of two or more groups of subjects. The assessment of differences 
between groups (intra-group measurement) has been an elusive entity. The focus 
of this discussion is the use of the Q-technique as an intra-group measure. 

Data from a curriculum study (Behm, 1975) are used herein as an example 
of the statistical procedures discussed. The data include responses from five 
faculty at each of six associate-degree nursing (ADN) programs and ten head 
nurses from each of six hospitals associated with these training programs. 
After the relative effectiveness of each ADN program was assessed by mailed 
^ questionnaire, the subjects performed a forced Q-sort of sixty nursing functions. 
^ The purpose of the Q-sort was to determine the agreement between ADN faculty and 
QQ head nurses relative to the training that nurses should receive. The results 

of the study supported the hypothesis that the faculty of the more effective 
lO programs would display closer agreement with the head nurses on the Q-sort than 
. would the faculty of programs perceived to be less effective. Those interested 



in a detailed discussion of these results are referred to the curriculum study. 

Typical Q-Sort Analysis 
Q-correlations, rank correlations across respondents, factor analysis 
which identify respondent loadings and item loadings are often used to analyze 
q-sort data. The coefficient of concordance (W) is also used to analyze these 
data. W is commonly used to determine the agreement among several sets of 
rankings (i.e. within or inter-group ag;reement) . The relationship between W 
and the Spearman rank correlation ' C^) permits the use of W as an intra-^group 
measure. 

The Relationship Between Spearnian Rank Order Correlations 
and The Coefficient cf Concordance 

The coefficient of concordance measures the comiQonality of judgment for 
"m" observers. The Spearman rank order correlation coefficient , a special 
case of the Pearson product moment correlation, is a generally accepted method 
of determining the relationship between sets of rankings such as Q-sort data. 
Both computations require a correction for tied ranks. 

The average Spearman ra.nk order correlation bears a linear relationship to 
the coefficient of concordance (Kendall, 1962, p. 95). 

Pave = mW-1 ^ Transposing^W = >^ave m -> ^ave + 1 

. ' m 

It is this linear relationship between /O and W which permits the inter- and 
intra-group analysis of Q-sort data which this paper describes. An advantage of 
using/> for computing W, that needs to be mentioned here, is that computer pro- 
grams such as the Statistical Package for Social Sciences (SPSS) include rank 
order correlations, but not the coefficient of concordance. 
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Inter- and Intra-group Agreement 
The data from Behm's study (mentioned earlier) yielded three correlation 
matrices. One matrix, instructors, was 5 x 5 with ten intercorrelations, 
another, nurses, was 10 x 10 x-zith 45 intercorrelations, and the combined 15 x 15 
matrix with 105 intercorrelations. The average correlation within the instructor 
matrix can be used to compute W and yield a measure cf the agreement among the 
instructors. The nurse matrix can be used in the same way to yield a measure 
of their inter-group agreement. However, the combined matrix cannot be used 
to assess the agreement between the groups since more than half of the inter-^ 
correlations represent inter-group agreement (see Figure 1) . 

(INSERT FIGURE 1 ABOUT HERE) 
The inter-group agreement was computed for each of the sets of data by 
using a direct computational formula (Siegel, 1956, p. 231) and by using the 
average of the rank correlations. A comparison of the results of these com- 
putations indicated that^^ proved to be a more conservative method and that the 
small discrepancies between the two methods increase as W increases. The small 
discrepancies result, in part, from the different treatment of ties in the 
coefficient of concordance and the Spearman rank order correlation. This 
comparison indicated that W could be accurately calculated for the curriculum 
study Q-sort data by either/? or a direct computatiohal formula. 

Given that W can be accurately calculated from yO. , it follows that the 
intra-«group agreement can be calculated from the residual intercorrelations (the 
rectangular area in Figure 1) . In our example the residual section which includes 
50 pairs of faculty and head nurses was indicative of this agreement. A rank 
order correlation was calculated for each nurse-faculty combination (50 for each 
program) , The average of these correlations was used to calculate a residual 
coefficient of concordance (Wj^) for the agreement between the nurses and faculty 
members. - 
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Intra-group W's were also calculated using the average of the 105 
correlations in the sample space- This procedure vould produce results 
equivalent to those produced by a direct computational formula. These 
results indicated different levels of agreement than those shown by the W's. 
For example > Program 1 and Program 2 displayed nearly identical levels of 
agreement based on the standard calculations (see Figure 2; W-j^ = ,46, 
^2 = -45). 

(INSERT FIGURE 2 ABOUT HERE) 
However, the W^'s which resulted from the^^of tha rectangular sample ^ 
space indicated that Program 2 was in greater accord ^^th its practice 
setting than was Program 1 (Wj^^ = .382, = -512). In this instance, the 
levels of agreement between the faculty members and head nurses were obscured 
by the inter-group agreements. In Program 1 the faculty members and nurses 
displayed a high level of agreement among themselves but displayed a low 
level of -agreement between the groups. The faculty and head nurses associated 
with Program 2 showed a lower level of inter-group agreement but a higher 
degree of intra--group agreement. Some programs displayed equal levels of inter- 
and intra-group agreement. In these instances, the two methods yielded 
equivalent results. However, the analysis of the curriculum study data and 
other sample data demonstrated the importance of focusing upon the residual 
agreement (Wj^) when the intent of the research is to examine the agreement 
between, groups of subjects performing a Q-sort. 

Conclusion 

The purpose of this article has been to present a technique for assessing 
both inter- and intra-group agreement of Q-sort responses. The technique 
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relies on the relationship between the average Spearman rank order correlation 

^^and the coefficient of concordance W. The use of permits the ex- 

amination of the agreement between groups of subjects performing a Q-sort. 
The direct computational method for W tends to obscure the level of intra- 
group agreement. 

In addition to providing an accurate assessment of intra-group agreement, 
the use of - permits the researcher to utilize computer programs such as 

. the Statistical Package of Social Sciences (SPSS) and to test the significance 
. of the different levels of agreement by treating each inter-correlation as 

a score (le. without regard to the distribution of the correlation coefficients) 
(Block, 1964). 

In summation, whenever the examination of the agreement between groups 
of subjects performing a Q-sort is of interest, the average of the inter- 
correlations should be used instead of the standard coefficient of concordance. 
This recommended treatment of the data can reveal a more accurate intra- 
group assessment than the traditional methods of analysis. 
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